Methods and systems for removing autofluorescence from images

ABSTRACT

Method and systems for removing any inherent autofluorescence associated with a biological material, comprising: acquiring a first reference image of the biological material; acquiring a first set of one or more images of the biological material using one or more filters corresponding to one or more information dyes; applying the one or more additional dyes to the biological material and then acquiring a second set of one or more images comprising a separate image of the biological material with each of the filters corresponding to the information dyes and a second reference image of the biological material; coregistering the first and second reference images; and then removing any inherent autofluorescence exhibited in the informational images acquired.

BACKGROUND

The invention relates generally to methods and systems for removinginherent autofluorescence of biological materials from images of thosebiological materials.

Tissue auto-fluorescence (AF) is a fundamental problem in microscopy andsurgical applications. It reduces the signal detection sensitivity, andin some cases may cause failure in the detection of fluorescent dyesignals. Accurate detection of protein-specific fluorescent dyes iscritical for many microscopy applications, as such molecular pathologyimaging, where quantitation of molecular pathways has significantimplications such as predicting drug response, therapy planning, andpopulation segmentation of cancer patients.

In recent years the development of numerous fluorescent dyes has madeoptical fluorescent microscopy the method of choice for diseaserecognition. Numerous studies have used fluorescent spectroscopytechniques to study the variations in tissue auto-fluorescence fordiagnosis of colorectal, breast, lung, cervical, colon, gastrointestinaltract, and cancer. However, these methods require extensive modeling oftissue-specific auto-fluorescence (AF) spectra. This tedious modelingprocess, which may not always be sufficient, can be side stepped byusing multiplexing techniques in which artificially introduced dyes ordyes are used to track specific proteins. Multiplexing involvesacquiring images of different dyes with non-overlapping emission orexcitation spectra through filter cubes that match the emission andexcitation spectra of each dye. However, in such methods, theprotein-specific fluorescence emitted by these dyes, upon appropriateexternal light excitation, is combined in unknown proportions with theinherent tissue autofluorescence (AF) signal, greatly reducing theirefficacy. Thus separation and removal of inherent tissue AF wouldgreatly improve the accuracy of such methods.

Although various strategies for tissue AF removal have been proposed andstudied in the literature, such as, using liquid crystal tunablefilters, fluorescence polarization, dual-wavelength differentialfluorescence correction, confocal laser scanning microscopy andtime-resolved fluorescence microscopy, many of these strategies make useof expensive multi-spectral imaging hardware, over the entire spectralrange, followed by spectral un-mixing. Apart from hardware augmentation,there are also various chemical processes that can be used to reduce theeffect of tissue AF.

Digitally acquired fluorescence microscope images can also be processedretrospectively using software methods, to separate tissue AF from therelevant dye fluorescence. Some of these methods rely on acquiringestimates of the pure AF signal and using them to remove AF from imagescontaining both dye and AF signals by a weighted subtraction. Others usestatistical correlation techniques to correct for the additive AFsignal. While these techniques are more cost effective than usingmulti-spectral imaging hardware, they may not be able to completelyremove the AF component from fluorescence microscopy images.

BRIEF DESCRIPTION

The methods and systems of the invention generally use a two-step imageacquisition process to remove the inherent tissue AF from fluorescentimages. These methods and systems may be used with or without chemicaland/or photobleaching AF reduction techniques.

Rather than acquiring images of all the dyes all at once using a set ofoptimum filter cubes tuned to specific dyes, the image acquisition iscarried out in two general steps. In the first step, a reference imageof the biological material is acquired. The reference image may beacquired by staining the biological material first with a reference dye,such as one or more low AF dyes (e.g. ultra-violet or infra-red), andthen a first reference image is acquired using a filter corresponding tothe low AF dye. The reference image may also be acquired withoutapplying a reference dye, by using a filter corresponding to a cyanfluorescent protein. A first set of images also comprises images takenwith filters corresponding to one or more additional dyes before suchdyes are applied. These images taken with the corresponding filters,except the low AF dyes, represent the tissue autofluorescence at theirspecific spectra. In the second step, the additional dyes are added, andthen separate images with each of the filters, including the referencefilter, are acquired. The reference images are coregistered by aligningthe first reference image with the second reference image using thestructures that are common in both steps. Then, a robust estimationprocess is used to separate the AF signal acquired in the first stepfrom the AF and dye signal acquired in the second step, resulting inimages that are free of AF. The methods and systems of the inventionhave the technical effect of preserving the signal while reducing oreliminating the AF, which increases the resulting signal-to-AF ratio andthe overall sensitivity of detection. These cost effective methods andsystems obviate the need for complicated and expensive instrumentationor chemical techniques. The methods and systems are adaptable to widearray of tissue micro arrays (TMA). However, they are not limited to usewith TMAs and may be applied to any fluorescence imaging application inwhich a reference low AF dye or probe and an additional dye orfluorescent probe are used for comparison, and successive images usingfilters corresponding to the reference dye and the active dye can betaken and coregistered. For the purpose of disclosing these methods andsystems, the term dye as used herein, encompasses fluorescent andnon-fluorescent imaging agents, and is used interchangeably in theexamples herein, but are not intended to be limiting in their scope oruse.

An embodiment of the method of the invention, for removing any inherentautofluorescence associated with a biological material, generallycomprises the steps of: a) acquiring a first reference image of thebiological material; b) acquiring a first set of one or more images ofthe biological material using one or more filters corresponding to oneor more information dyes; c) applying the one or more additional dyes tothe biological material and then acquiring a second set of one or moreimages comprising a separate image of the biological material with eachof the filters corresponding to the information dyes and a secondreference image of the biological material; d) coregistering the firstand second reference images; and e) then removing any inherentautofluorescence exhibited in the images acquired in steps c).

The first and second reference images may be acquired using a filtercorresponding to a cyan fluorescent protein; and/or wherein step a)further comprises the step of, applying a reference dye, having a highsignal to autofluorescence ratio, to the biological material and whereinthe first and second reference images are acquired using a filtercorresponding to the reference dye. The reference dye may comprise a dyethat corresponds to the UV spectrum, such as, but not limited to DAPI,and/or a dye that corresponds to the IR spectrum.

The first reference image may be a fixed image having a coordinatesystem and second reference image is a moving image having a coordinatesystem, and wherein the reference images are coregistered at least inpart, to form a composite having a coordinate system, by estimating oneor more underlying transformation parameters to map the moving imageonto the fixed image coordinate system. The reference images may becoregistered, at least in part, using a similarity transform thatincorporates translation, rotation, and scaling.

An embodiment of the system of the invention for removing inherentautofluorescence associated with a biological material, generallycomprises: a) a processing device adapted to analyze a reference imageof a biological material that exhibits the presence of one or morereference dyes, having a high signal to autofluorescence ratio; b) oneor more filters corresponding to the reference dye and one or morefilters corresponding to one or more information dyes; c) a digitalimaging device adapted to acquire a first set of one or more images ofthe biological material, in conjunction with the filters correspondingto the reference dye and the one or more information dyes; and furtheradapted to acquire a second set of one or more images of the biologicalmaterial, wherein the biological material further exhibits the presenceof one or more of the information dyes, in conjunction with the filterscorresponding to the reference dye and the one or more information dyes;wherein the second set of images comprises a separate image of thebiological material with each of the filters corresponding to thereference dye and the one or more information dyes; and wherein theprocessing device is further adapted to coregister the reference images;and then remove the inherent autofluorescence, from the imagesexhibiting the presence of one or more of the information dyes. If areference dye is used, at least one of the reference dyes may correspondto the UV spectrum and/or the IR spectrum.

The digital imaging device may be adapted to acquire a reference imageof the biological material using a reference filter corresponding to acyan fluorescent protein; wherein the processor is adapted to coregisterthe reference images by estimating one or more underlying transformationparameters to map the image coordinate system. The reference images maybe coregistered using, but not necessarily limited to, one or moreintensity-based or featured based parameters. The reference images maybe coregistered, at least in part, using a similarity transform thatincorporates translation, rotation, and scaling.

Another embodiment of the system of the invention for removing inherentautofluorescence associated with a biological material, generallycomprises: a) a processing device adapted to analyze a reference imageof a biological material taken using a reference filter corresponding toa cyan fluorescent protein, having a high signal to autofluorescenceratio; b) one or more filters corresponding to one or more informationdyes; c) a digital imaging device adapted to acquire a first set of oneor more images of the biological material, in conjunction with thereference filter and the filters corresponding to the one or moreinformation dyes; and further adapted to acquire a second set of one ormore images of the biological material, wherein the biological materialfurther exhibits the presence of one or more of the information dyes, inconjunction with the reference filters and the filters corresponding tothe one or more information dyes; wherein the second set of imagescomprises a separate image of the biological material with the referencefilter and with each of the filters corresponding to the one or moreinformation dyes; and wherein the processing device is further adaptedto coregister the reference images; and then remove the inherentautofluorescence, from the images exhibiting the presence of one or moreof the information dyes. Similarly, the reference images may becoregistered using, but not limited to, an intensity-based or a featuredbased parameters. The reference images may also be coregistered, atleast in part, using a similarity transform that incorporatestranslation, rotation, and scaling.

DRAWINGS

These and other features, aspects, and advantages of the presentinvention will become better understood when the following detaileddescription is read with reference to the accompanying drawings in whichlike characters represent like parts throughout the drawings, wherein:

FIG. 1 a shows a joint distribution of the first acquisition and thesecond acquisition intensities calculated using all images on the TMA.The first acquisition comprises the image of the AF, and the secondacquisition is after the dye is added. The line shows the estimatedinitial line.

FIG. 1 b shows a joint distribution of the first acquisition and thesecond acquisition intensities. The line represents the final fittedline after the algorithm converges.

FIG. 2 shows six images of the first and second image acquisitions usingvarious dyes. The top row comprises images from the first imageacquisition. The middle row comprises images from the second imageacquisition. The bottom row comprises images that are corrected andAF-free. The images in the left column comprise Cy5 dye directlyconjugated to Pan-Cadherin, a membrane protein. The images in the rightcolumn comprise Cy3 dye directly conjugated to Estrogen Receptor. Thearrows indicate tissue regions in the image, such as blood cells andfat, which generally exhibit high AF, in which the AF has been removed.

FIG. 3 is a schematic diagram of an embodiment of the automated systemfor carrying out the methods.

DETAILED DESCRIPTION

The methods and systems of the invention comprise a two-step imageacquisition process to enhance the accuracy of multiplexing and otherfluorescent imaging of biological material. In the first step, areference image of the biological material is acquired. This image maybe acquired simply by using a filter corresponding to a cyan fluorescentprotein, or, the tissue may be stained first with a dye or fluorescentimaging agent that has the highest signal to autofluorescence ratio (lowAF) and then an image of the tissue stained with the low AF dye,otherwise referred to herein as a reference dye, is taken with a filtercube corresponding to the low AF dye. In addition to the image throughthe filter cube that corresponds to the low AF dye, otherwise referredto herein as a reference image, additional images are taken using filtercubes corresponding to one or more additional dyes or fluorescentimaging agents. These additional dyes, otherwise referred to herein asinformation dyes, are not limited in type and include any dye or imagingagent capable of illuminating, enhancing or activating anycharacteristic or feature of a biological tissue. The terms “additional”and “information”, in conjunction with the term “dye”, are used hereinmerely to distinguish from the reference dye or dyes that may be used toacquire the reference image.

The reference image or the first set of images need not necessarily betaken in immediate or close temporal or physical proximity to the timeand location of the second set of images. Rather these images may betaken and stored for later retrieval or access. However, the first setof images must be configured and stored so that the reference image ofthe first set of images can be coregistered with the reference imagefrom the second set of images by using the methods and systems hereinfor estimating one or more underlying transformation parameters to mapthe image coordinate system.

The reference image may also be an image taken after a photobleachingstep. For example, the methods and systems are also contemplated for usein multiplexing applications. In multiplexing applications, a pluralityof dyes (typically corresponding to different channels) and theacquisition of the image for each dye (or channel) are applied and takenin successive rounds, in between which, a photobleaching agent isapplied to the biological material so that another dye can besubsequently applied in a successive round of dye application and imageacquisition. In such applications, an image of the biological materialtaken after photobleaching, with an appropriate filter, may serve as areference image. If more than one reference image is acquired in a givenmultiplexing process, one or more of these reference images may be usedin the co-registering step.

After the first set of images is acquired, the tissue is removed fromthe imaging device, such as, but not limited to, a microscope, and thenthe information dyes are applied. As noted, these additional dyes areused to elucidate, enhance or modify one or more features orcharacteristics of the biological material to gather information from orabout the biological material. A second set of images using the entireset of filters, both reference filters and filters corresponding to theadditional dyes, is acquired again. The two reference images areregistered using the common channel (corresponding to the cyanfluorescent protein or low AF reference dye) between the two steps byestimating one or more underlying transformation parameters to map thecomposite moving image onto the composite fixed image coordinate system.The AF of the biological material is then removed from the image.

These methods and systems may be used to image and analyze a biologicalsample to discern inter alia the presence, absence, concentration,and/or spatial distribution of one or more biological materials ortargets in a biological sample or tissue.

To more clearly and concisely describe and point out the subject matterof the claimed invention, the following definitions are provided forspecific terms, which are used in the following description and theappended claims.

As used herein, the term “biological material” refers to materialobtained from, or located in, a biological subject, including sample ofbiological tissue or fluid origin obtained in vivo or in vitro andbiological materials that may be located in situ. Such samples can be,but are not limited to, body fluid (e.g., blood, blood plasma, serum, orurine), organs, tissues, fractions, and cells isolated from, or locatedin, mammals including, humans. Biological samples also may includesections of the biological sample including tissues (e.g., sectionalportions of an organ or tissue). Biological samples may also includeextracts from a biological sample, for example, an antigen from abiological fluid (e.g., blood or urine).

As used herein, the term “in situ” generally refers to an eventoccurring in the original location, for example, in intact organ ortissue or in a representative segment of an organ or tissue. In someembodiments, in situ analysis of targets may be performed on cellsderived from a variety of sources, including an organism, an organ,tissue sample, or a cell culture. In situ analysis provides contextualinformation that may be lost when the target is removed from its site oforigin. Accordingly, in situ analysis of targets describes analysis oftarget-bound probe located within a whole cell or a tissue sample,whether the cell membrane is fully intact or partially intact wheretarget-bound probe remains within the cell. Furthermore, the methodsdisclosed herein may be employed to analyze targets in situ in cell ortissue samples that are fixed or unfixed.

A biological material may include any material regardless of itsphysical condition, such as, but not limited to, being frozen or stainedor otherwise treated. In some embodiments, a biological material mayinclude a tissue sample, a whole cell, a cell constituent, a cytospin,or a cell smear. In some embodiments, a biological material may includea tissue sample. In other embodiments, a biological material may be anin situ tissue target, if successive images of the targeted tissue canbe obtained, first with the reference dye and subsequently with theadditional dyes. A tissue sample may include a collection of similarcells obtained from a tissue of a biological subject that may have asimilar function. In some embodiments, a tissue sample may include acollection of similar cells obtained from a tissue of a human. Suitableexamples of human tissues include, but are not limited to, (1)epithelium; (2) the connective tissues, including blood vessels, boneand cartilage; (3) muscle tissue; and (4) nerve tissue. The source ofthe tissue sample may be solid tissue obtained from a fresh, frozenand/or preserved organ or tissue sample or biopsy or aspirate; blood orany blood constituents; bodily fluids such as cerebral spinal fluid,amniotic fluid, peritoneal fluid, or interstitial fluid; or cells fromany time in gestation or development of the subject. In someembodiments, the tissue sample may include primary or cultured cells orcell lines.

In some embodiments, a biological sample includes tissue sections fromhealthy or diseases tissue samples (e.g., tissue section from colon,breast tissue, prostate). A tissue section may include a single part orpiece of a tissue sample, for example, a thin slice of tissue or cellscut from a tissue sample. In some embodiments, multiple sections oftissue samples may be taken and subjected to analysis, provided themethods disclosed herein may be used for analysis of the same section ofthe tissue sample with respect to at least two different targets (atmorphological or molecular level). In some embodiments, the same sectionof tissue sample may be analyzed with respect to at least four differenttargets (at morphological or molecular level). In some embodiments, thesame section of tissue sample may be analyzed with respect to greaterthan four different targets (at morphological or molecular level). Insome embodiments, the same section of tissue sample may be analyzed atboth morphological and molecular levels.

As used herein, the term “fluorescent imaging agent” refers fluorophoresthat are chemical compounds, which when excited by exposure to aparticular wavelength of light, emit light at a different wavelength.Fluorophores may be described in terms of their emission profile, or“color.” Green fluorophores (for example Cy3, FITC, and Oregon Green)may be characterized by their emission at wavelengths generally in therange of 515-540 nanometers. Red fluorophores (for example Texas Red,Cy5, and tetramethylrhodamine) may be characterized by their emission atwavelengths generally in the range of 590-690 nanometers. Examples offluorophores include, but are not limited to,4-acetamido-4′-isothiocyanatostilbene-2,2′disulfonic acid, acridine,derivatives of acridine and acridine isothiocyanate,5-(2′-aminoethyl)aminonaphthalene-1-sulfonic acid(EDANS),4-amino-N-[3-vinylsulfonyl)phenyl]naphthalimide-3,5 disulfonate(Lucifer Yellow VS), N-(4-anilino-1-naphthyl)maleimide, anthranilamide,Brilliant Yellow, coumarin, coumarin derivatives,7-amino-4-methylcoumarin (AMC, Coumarin 120),7-amino-trifluoromethylcouluarin (Coumaran 151), cyanosine;4′,6-diaminidino-2-phenylindole (DAPI),5′,5″-dibromopyrogallol-sulfonephthalein (Bromopyrogallol Red),7-diethylamino-3-(4′-isothiocyanatophenyl)-4-methylcoumarin, -,4,4′-diisothiocyanatodihydro-stilbene-2,2′-disulfonic acid,4,4′-diisothiocyanatostilbene-2,2′-disulfonic acid,5-[dimethylamino]naphthalene-1-sulfonyl chloride (DNS, dansyl chloride),eosin, derivatives of eosin such as eosin isothiocyanate, erythrosine,derivatives of erythrosine such as erythrosine B and erythrosinisothiocyanate; ethidium; fluorescein and derivatives such as5-carboxyfluorescein (FAM), 5-(4,6-dichlorotriazin-2-yl)aminofluorescein (DTAF),2′7′-dimethoxy-4′5′-dichloro-6-carboxyfluorescein (JOE), fluorescein,fluorescein isothiocyanate (FITC), QFITC (XRITC); fluorescaminederivative (fluorescent upon reaction with amines); IR144; IR1446;Malachite Green isothiocyanate; 4-methylumbelliferone; orthocresolphthalein; nitrotyrosine; pararosaniline; Phenol Red,B-phycoerythrin; o-phthaldialdehyde derivative (fluorescent uponreaction with amines); pyrene and derivatives such as pyrene, pyrenebutyrate and succinimidyl 1-pyrene butyrate; Reactive Red 4 (Cibacron™Brilliant Red 3B-A), rhodamine and derivatives such as6-carboxy-X-rhodamine (ROX), 6-carboxyrhodamine (R6G), lissaminerhodamine B sulfonyl chloride, rhodamine (Rhod), rhodamine B, rhodamine123, rhodamine X isothiocyanate, sulforhodamine B, sulforhodamine 101and sulfonyl chloride derivative of sulforhodamine 101 (Texas Red);N,N,N′,N′-tetramethyl-6-carboxyrhodamine (TAMRA); tetramethyl Rhodamine,tetramethyl rhodamine isothiocyanate (TRITC); riboflavin; rosolic acidand lathanide chelate derivatives, quantum dots, cyanines, pyreliumdyes, and squaraines.

For applications that additionally use probes, as used herein, the term“probe” refers to an agent having a binder and a label, such as a signalgenerator or an enzyme. In some embodiments, the binder and the label(signal generator or the enzyme) are embodied in a single entity. Thebinder and the label may be attached directly (e.g., via a fluorescentmolecule incorporated into the binder) or indirectly (e.g., through alinker, which may include a cleavage site) and applied to the biologicalsample in a single step. In alternative embodiments, the binder and thelabel are embodied in discrete entities (e.g., a primary antibodycapable of binding a target and an enzyme or a signal generator-labeledsecondary antibody capable of binding the primary antibody). When thebinder and the label (signal generator or the enzyme) are separateentities they may be applied to a biological sample in a single step ormultiple steps. As used herein, the term “fluorescent probe” refers toan agent having a binder coupled to a fluorescent signal generator.

For applications that require fixing a biological material on a solidsupport, as used herein, the term “solid support” refers to an articleon which targets present in the biological sample may be immobilized andsubsequently detected by the methods disclosed herein. Targets may beimmobilized on the solid support by physical adsorption, by covalentbond formation, or by combinations thereof. A solid support may includea polymeric, a glass, or a metallic material. Examples of solid supportsinclude a membrane, a microtiter plate, a bead, a filter, a test strip,a slide, a cover slip, and a test tube. In those embodiments, in which abiological material is adhered to a membrane, the membrane material maybe selected from, but is not limited to, nylon, nitrocellulose, andpolyvinylidene difluoride. In some embodiments, the solid support maycomprise a plastic surface selected from polystyrene, polycarbonate, andpolypropylene.

The methods and systems may be adapted for, but are not limited to, usein analytical, diagnostic, or prognostic applications such as analytedetection, histochemistry, immunohistochemistry, or immunofluorescence.In some embodiments, the methods and systems may be particularlyapplicable in histochemistry, immunostaining, immunohistochemistry,immunoassays, or immunofluorescence applications. In some embodiments,the methods and systems may be particularly applicable in immunoblottingtechniques, for example, western blots or immunoassays such asenzyme-linked immunosorbent assays (ELISA).

IMAGE ACQUISITION EXAMPLE

This example, although not limiting, acquires images of biologicalmaterial stained with DAPI, Cy3, and Cy5. Among these three dyes, DAPI,a nuclear marker, is known to have very high signal to autofluorescenceratio (low AF). In the first step, an image of the material using afilter cube corresponding to DAPI is acquired. Additionally, a first setof images of the material using filter cubes corresponding to Cy3 andCy5 is also acquired. These additional images using filter cubes imagescorresponding to the additional dyes, before the additional dyes areapplied, serve as reference or control images to the images taken afterthe additional dyes are applied which will then exhibit theautofluorescence corresponding to the additional dyes. Furthermore, anadditional, optional image using a filter cube that has minimal crosstalk with the DAPI, Cy3, and Cy5 is also acquired. This minimal crosstalk filter cube corresponds to a cyan fluorescent protein (CFP). Thisfilter cube is used when acquiring the image of the autofluorescenceonly, and is used only for image registration, as described furtherbelow.

After the first set of images is acquired, the additional dyes areapplied to the tissue. Then a second set of images using all of thefilter cubes (DAPI, CFP, Cy3, and Cy5) are acquired. Note that the DAPIand CFP images in both acquisitions are essentially the same, and can beused to determine the transformation, described below, that aligns thetwo sets of images. Once the images are registered, using the stepsdescribed below, the autofluorescent images acquired through the Cy3 andCy5 cubes in the first step are removed from the registered Cy3 and Cy5images in the second step.

Image registration algorithms are generally grouped into two categories:intensity-based and featured based. Feature extraction based algorithmstypically require an initial image analysis and segmentation step. Forpathology images, for example, the location, size, and shape of thenuclei can be extracted from DAPI stained images. This information canthen be used to align the images using a point matching technique.Features from epithelial tissue, stromal tissue, and glands/backgroundcan be extracted from the auto-florescence images as well, however it isgenerally more challenging to detect feature points consistently inthese images.

In this example embodiment, an intensity based registration method isused that does not require any prior segmentation information, and isapplicable to a broad class of dyes.

The DAPI image and the tissue AF image, acquired using a CFP filtercube, are added to form a composite image of the first set of images,which is referred to as the fixed image. While the AF image provideslarge-scale/global information, which is critical for the convergence ofthe registration algorithms, the DAPI image provides fine structuresessential for the accuracy of the registration. The fixed image isdenoted by I_(F)(x_(F),y_(F)). The fixed image is used to define thereference coordinate system of the composite image from the firstacquisition. The moving image is denoted by I_(M)(x_(M),y_(M)), which isthe composite image of the second set of images.

Registration is the estimation of the underlying transformationparameters that map the moving image onto the fixed image coordinatesystem, obtained as arguments that minimize a cost function F;

$\begin{matrix}{{\underset{\theta}{\arg \; \min}{F\left( {{I_{F}\left( {x_{F},y_{F}} \right)},{I_{M}\left( {T\left( {x_{M},{y_{M};\theta}} \right)} \right)}} \right)}},} & (1)\end{matrix}$

-   -   arg minF(I_(F)(x_(F),y_(F)),I_(M)(T(x_(M),y_(M);θ))) (1)        where, T represents spatial transformation with parameters θ.        More specifically, a similarity transform is used in this        embodiment that incorporates translation, rotation, and scaling.        The translation and rotation try to correct the misplacement of        the tissue slide, and scaling can handle distortions due to        small focal plane changes. This transformation maps the moving        image into the fixed image coordinate system;

$\begin{matrix}{{T\left( {x_{M},{y_{M};\theta}} \right)} \equiv {{\begin{bmatrix}\theta_{1} & \theta_{2} \\{- \theta_{2}} & \theta_{1}\end{bmatrix}\begin{pmatrix}x_{M} \\y_{M}\end{pmatrix}} + {\begin{pmatrix}\theta_{3} \\\theta_{4}\end{pmatrix}.}}} & (2)\end{matrix}$

Note that higher order transformation models, such as affine or higherorder polynomial transformations, can be used if geometric lensdistortion is a factor.

There are a great number of measures that can be used as the costfunction F, such as, but not limited to, mean-square-error, crosscorrelation, Kullback-Liebler distance, gradient difference metric, andmutual information. Due to its demonstrated robustness in multi-modalityimage registration, the negative of mutual information (MI), is used inthis example as the cost function. MI is defined as;

$\begin{matrix}\begin{matrix}{{F\left( {{I_{F}\left( {x_{F},y_{F}} \right)},{I_{M}\left( {T\left( {x_{M},{y_{M};\theta}} \right)} \right)}} \right)} =} \\{{- {H\left( {I_{F}\left( {x_{F},y_{F}} \right)} \right)}} - {H\left( {I_{M}\left( {T\left( {x_{M},{y_{M};\theta}} \right)} \right)} \right)}} \\{+ {H\left( {{I_{F}\left( {x_{F},y_{F}} \right)},{I_{M}\left( {T\left( {x_{M},{y_{M};\theta}} \right)} \right)}} \right)}}\end{matrix} & (3)\end{matrix}$

where H represents the entropy of the image. Here MI is negated tofacilitate the minimization process defined in the first equation.

After the first set of images, referred to in this example embodiment asthe fixed image, is registered with the second set of images, referredto in this example embodiment as the moving image, the inherent tissueautofluorescence is removed. In this example, the imaging process ismodeled and a robust estimation method is used to estimate the modelparameters.

As noted, the two-step acquisition provides two images: one AF onlyimage, and one AF plus dye signal image. In this example, a robustregression method is used to compute the Dye Signal image. The image ofthe AF acquired in the first step is denoted as F(x,y), and the secondacquired image (Signal plus AF) is denoted as S(x,y). These two imagesare related by the following relation,

$\begin{matrix}{{S\left( {x,y} \right)} = \left\{ \begin{matrix}{{D\left( {x,y} \right)} + {\alpha \; {F\left( {x,y} \right)}} + \beta} & {{if}\mspace{14mu} {there}\mspace{14mu} {is}\mspace{14mu} {dye}\mspace{11mu} {staining}\mspace{14mu} {at}\; \left( {x,y} \right)} \\{{\alpha \; {F\left( {x,y} \right)}} + \beta} & {{otherwise},}\end{matrix} \right.} & (4)\end{matrix}$

where D(x,y) is the unknown dye image, α is the relative gain constant,and β is the relative offset. If the images are normalized to subtractthe dark current, β can be set to zero. If the exposure times are knownand the excitation light intensity has not changed between the twoacquisitions, α can be set to the ratio of the exposure times betweenthe two acquisitions. Note that due to spectral leakage and reflectionsin the optical system, the observed images typically have an offsetcomponent that is a function of the exposure time. Since the exposuretimes between the first acquisition and the second acquisition areusually different, the relative offset may be modeled as a separateterm, rather than incorporating with the unknown term D(x,y). Byexplicitly parametrizing the relative offset term, the non-negativityconstraint on D(x,y) can be imposed, and the cost function adaptedaccordingly.

Although various source separation methods are known, such asStatistical decorrelation, Principal Component Analysis, IndependentComponent Analysis (ICA), and non-negative ICA, all of these methodsestimate a coordinate transformation matrix which transforms F(x,y) andS(x,y) into a new coordinate system such that they are as muchuncorrelated as possible (PCA), or as independent as possible (ICA).However a close look at the joint distributions of F(x,y) and S(x,y)show that there is no such transform that can achieve fully uncorrelatedor independent components. For example, FIG. 1 a shows the jointdistribution computed from multiple images on the same TMA. The areaabove the line indicates the expression and the line itself indicatesthe autofluorescence. Note that the distribution is formed by twoclusters; one cluster corresponding to pixels are AF on both images, andthe cluster that is a signal image on one image and AF on the otherimage. Since these two clusters are not orthogonal, and their proportionwith respect to each other can significantly change from tissue totissue, PCA and ICA type methods are generally insufficient. A methodthat does not require the transformation to be orthogonal is theNon-negative Matrix Factorization (NMF). However this method isgenerally prone to local minima, and good performance can usually onlybe achieved with overconstraint systems. Instead, the preferred methodsand systems of the invention use a robust regression method to solve theunknown transformation parameters (α,β), and the unknown signal(D(x,y)).

The unknown constants are estimated by treating the D(x,y) as outliersand solving the following robust cost function

$\begin{matrix}{{\left( {\overset{\Cap}{\alpha},\overset{\Cap}{\beta}} \right) = {\underset{\alpha,\beta}{argmin}{\sum\limits_{x,y}{\rho \left( {{S\left( {x,y} \right)} - {\alpha \; {F\left( {x,y} \right)}} - \beta} \right)}}}},} & (5)\end{matrix}$

where ρ is a Huber's robust cost function [34,35], defined as,

$\begin{matrix}{{\rho (r)} = \left\{ \begin{matrix}{r^{2}/2} & {r \leq k} \\{{kr} - {k^{2}{r/2}}} & {r > {k.}}\end{matrix} \right.} & (6)\end{matrix}$

Note that ρ is slightly different than traditional symmetric Huber costfunction [34]. It is a simple least squares cost function for r≦k,rather than for r≦|k|. Since D(x,y) is a non-negative function, thereare no negative outliers that are significantly smaller than theexpected linear form. This one sided non-symmetric cost function addsmore cost for negative D(x,y), and it is essential for the convergenceof the algorithm by biasing towards the desired solution. Equation 6 canbe solved by iterative weighted least squares (IRLS),

$\begin{matrix}\begin{matrix}{\left( {\overset{\Cap}{\alpha},\overset{\Cap}{\beta}} \right) = {\underset{\alpha,\beta}{argmin}{\sum\limits_{x.y}{{w\left( {x,y} \right)}\left( {{S\left( {x,y} \right)} - {\alpha \; {F\left( {x,y} \right)}} - \beta} \right)^{2}}}}} \\{= {{\underset{\alpha,\beta}{argmin}\left( {\overset{->}{S} - {\left( {\overset{->}{F},\overset{->}{1}} \right)\left( {\alpha,\beta} \right)^{T}}} \right)}{W\left( {\overset{->}{S} - {\left( {\overset{->}{F},\overset{->}{1}} \right)\left( {\alpha,\beta} \right)^{T}}} \right)}^{T}}}\end{matrix} & (7) \\{{where},} & \; \\{{w(r)} = {\frac{\rho (r)}{r} = \left\{ \begin{matrix}1 & {r \leq k} \\{k/r} & {r > {k.}}\end{matrix} \right.}} & (8)\end{matrix}$

The IRLS solution can then be implemented by iterating the followingequations,

({circumflex over (α)}_(k),{circumflex over (β)}k)^(T)=(({right arrowover (F)},{right arrow over (1)})^(T) W _(k-1)({right arrow over(F)},{right arrow over (1)})⁻¹({right arrow over (F)},{right arrow over(1)})^(T) W _(k-1) {right arrow over (S)}  (9)

where

W _(k-1)=diag(w(S(x,y)−{circumflex over (α)}F(x,y)−{circumflex over(β)}_(k-1))).  (10)

FIG. 1 a shows S(x,y) versus F(x,y) for all the images on the same TMA.Notice that the joint distribution is composed of two clusters, oneformed by corresponding AF on both images, and one formed by AF and DyeSignal. The line shown in FIG. 1 a shows the initial estimate for (α,β),estimated by fitting a line to the lower bound of the jointdistribution. Let (f_(i),s_(i)) are the intensity values on the firstand second images such that for all f_(i)=F(x,y),

$\begin{matrix}{s_{i} = {\min\limits_{{({x^{\prime},y^{\prime}})} \in {\{{{({x,y})}:{{({f_{i} - R})} < {F{({x,y})}} \leq {({f_{i} + R})}}}\}}}{S\left( {x^{\prime},y^{\prime}} \right)}}} & (11)\end{matrix}$

where R is set to (Dynamic Range)/256. The initial parameter values canthen be estimated by least squares regression by minimizing;

$\begin{matrix}{\left( {{\overset{\Cap}{\alpha}}_{0},{\overset{\Cap}{\beta}}_{0}} \right) = {\underset{\alpha,\beta}{argmin}{\sum\limits_{i}{\left( {s_{i} - {\alpha \; f_{i}} - \beta} \right)^{2}.}}}} & (12)\end{matrix}$

The scale factor, k in Equations 6&8, is estimated by first determiningthe local standard deviations by Median Absolute Deviation (MAD);

$\begin{matrix}{{k_{i} = {1.345\underset{{({x^{\prime},y^{\prime}})} \in {\{{{({x,y})}:{{({f_{i} - R})} < {F{({x,y})}} \leq {({f_{i} + R})}}}\}}}{median}{{{S\left( {x^{\prime},y^{\prime}} \right)} - {\underset{({x^{\prime},y^{\prime}})}{median}\left( {{S\left( {x^{\prime},y^{\prime}} \right)} - {{\overset{\Cap}{\alpha}}_{0}f_{i}} - {\overset{\Cap}{\beta}}_{0}} \right)}}}}},} & (13)\end{matrix}$

and then taking the median of all the local MAD estimates,

{circumflex over (k)}=median(k _(i)).  (14)

Both of the above equations are well known to be robust to outliers.

FIG. 1 a shows the joint distribution of F(x,y) and S(x,y) computed fromall images on the same TMA. The line on FIG. 1 a shows the initial modelparameters computed by using Equation 12. Beginning with the initialparameter values, equations 9&10 refine the parameter estimates for eachindividual image on the TMA. This refinement is capable of handlingslight tissue and AF variations among different tissues, particularlynormal versus metastasis, or expressed versus non-expressed markers.FIG. 1 b shows the joint distribution and the final linear parametersfor the image show in FIG. 2 (Right column).

EXAMPLE

In this example, TMAs of breast tissue were stained with DAPI, Cy3, andCy5. DAPI is a nuclear marker that binds to DNA. DAPI was used in thisexample, for image registration and to quantify percentage expression ofnuclear related proteins. Cy3 is directly conjugated withanti-pan-cadherin antibody, a membrane protein, and Cy5 is directlyconjugated with anti-Estrogen Receptor (ER) antibody. ER may or may notbe expressed depending on if the patient is ER+ or ER−. In this example,expression of ER correlates with response to anti-estrogen treatment(tamoxifen or others) or chemotherapy, and is associated withbetter-differentiated tumors. In this example, the ER expression isdifferentiated from AF, particularly from blood cells and fat that showsup as bright structures, because ER expression may be readily confusedwith real expression. FIG. 2 (Right Column) shows the images of Cy5channel at the first step (Top), at the second step (Middle), and thecorrected image (Bottom). The arrows show the region with high AF, whichmay be due to bloods cells or fat. The AF is removed while preservingthe ER expression. FIG. 2 (Left Column) shows the images of the Cy3channel. The arrows show the cleaned high AF structures. In addition toremoving high-AF structures, low-AF structures are reduced as well.

The methods preserve the signal while reducing the AF; hence thesignal-to-AF ratio is increased. High-AF structures such as fat andblood cells are also successfully removed, which is critical todifferentiate specific protein expression from non-specific expression.Removing these structures also enables accurate automated proteinexpression.

The automated system 10 (FIG. 3) for carrying out the methods generallycomprises: a storage device 12 for at least temporarily storing one ormore images of one or more biological materials; and a processor 14adapted to analyze a reference image of a biological material thatexhibits the presence of a cyan fluorescent protein and/or one or morereference dyes, having a high signal to autofluorescence ratio; one ormore filters corresponding to the cyan fluorescent protein and/orreference dyes and one or more filters corresponding to one or moreadditional dyes; and a digital imaging device adapted to acquire thefirst set of one or more images of the biological material, inconjunction with the reference filters and the filters corresponding tothe one or more additional dyes; and further adapted to acquire a secondset of one or more images of the biological material, wherein thebiological material further exhibits the presence of one or more of theadditional dye, in conjunction with the reference filters and thefilters corresponding to the one or more additional dyes; wherein thesecond set of images comprises a separate image of the biologicalmaterial with each of the reference filters and the filterscorresponding to the additional dyes.

Processing device 14 is further adapted to coregister the referenceimage; and then remove the autofluorescence, associated with thereference images, from one or more of the images.

The storage device may comprise, but is not necessarily limited to, anysuitable hard drive memory associated with the processor such as the ROM(read only memory), RAM (random access memory) or DRAM (dynamic randomaccess memory) of a CPU (central processing unit), or any suitable diskdrive memory device such as a DVD or CD, or a zip drive or memory card.The storage device may be remotely located from the processor or themeans for displaying the images, and yet still be accessed through anysuitable connection device or communications network including but notlimited to local area networks, cable networks, satellite networks, andthe Internet, regardless whether hard wired or wireless. The processoror CPU may comprise a microprocessor, microcontroller and a digitalsignal processor (DSP).

The storage device 12 and processor 14 may be incorporated as componentsof an analytical device such as an automated high-throughput system thatstains and images the TMAs in one system and still further analyzes theimages for any number of applications such as, but not limited to,diagnostic applications. One of more of these steps may be configuredinto one system or embodied in one or more stand-alone systems. System10 may further comprise a means for displaying 16 one or more of theimages; an interactive viewer 18; a virtual microscope 20; and/or ameans for transmitting 22 one or more of the images or any related dataor analytical information over a communications network 24 to one ormore remote locations 26.

The means for displaying 16 may comprise any suitable device capable ofdisplaying a digital image such as, but not limited to, devices thatincorporate an LCD or CRT. The means for transmitting 22 may compriseany suitable means for transmitting digital information over acommunications network including but not limited to hardwired orwireless digital communications systems. The system may further comprisean automated device 28 for applying one or more of the stains and adigital imaging device 30 such as, but not limited to, an imagingmicroscope comprising an excitation source 32 and capable of capturingdigital images of the TMAs. Such imaging devices are preferably capableof auto focusing and then maintaining and tracking the focus feature asneeded throughout processing.

These methods and systems are not limited to any specific imagingagents, morphological dyes, biomarkers or probes. Any fluorescent ornonfluorescent dye or imaging agent that enables some informative aspector feature of a biological material to be actually or artificiallyvisualized so that it can be digitally imaged and processed, would besuitable. Suitable dyes and imaging agents include, but are notnecessarily limited to, cytological or morphological dyes, immunologicaldyes such as immunohisto- and immunocyto-chemistry dyes, cytogeneticaldyes, in situ hybridization dyes, cytochemical dyes, DNA and chromosomemarkers, and substrate binding assay dyes.

While only certain features of the invention have been illustrated anddescribed herein, many modifications and changes will occur to thoseskilled in the art. It is, therefore, to be understood that the appendedclaims are intended to cover all such modifications and changes as fallwithin the true spirit of the invention.

1. A method for removing any inherent autofluorescence associated with abiological material, comprising the steps of: a) acquiring a firstreference image of the biological material; b) acquiring a first set ofone or more images of the biological material using one or more filterscorresponding to one or more information dyes; c) applying the one ormore additional dyes to the biological material and then acquiring asecond set of one or more images comprising a separate image of thebiological material with each of the filters corresponding to theinformation dyes and a second reference image of the biologicalmaterial; d) coregistering the first and second reference images; e) andthen removing any inherent autofluorescence exhibited in the imagesacquired in steps c).
 2. The method of claim 1, wherein the first andsecond reference images are acquired using a filter corresponding to acyan fluorescent protein.
 3. The method of claim 1, wherein step a)further comprises the step of, applying a reference dye, having a highsignal to autofluorescence ratio, to the biological material and whereinthe first and second reference images are acquired using a filtercorresponding to the reference dye.
 4. The method of claim 3, whereinthe dye applied in step a) comprises a dye that corresponds to the UVspectrum.
 5. The method of claim 4, wherein the dye applied in step a)corresponding to the UV spectrum is DAPI.
 6. The method of claim 3,wherein the dye applied in step a) comprises a dye that corresponds tothe IR spectrum.
 7. The method of claim 1, wherein the first and secondreference images of the biological material are acquired using a filtercorresponding to a cyan fluorescent protein; and wherein the firstreference image is a fixed image having a coordinate system and secondreference image is a moving image having a coordinate system, andwherein the reference images are coregistered at least in part, to forma composite having a coordinate system, by estimating one or moreunderlying transformation parameters to map the moving image onto thefixed image coordinate system.
 8. The method of claim 1, wherein step a)further comprises the step of, applying a reference dye, having a highsignal to autofluorescence ratio, to the biological material; whereinthe first and second reference images are acquired using a filtercorresponding to the reference dye; and wherein the first referenceimage is a fixed image having a coordinate system and second referenceimage is a moving image having a coordinate system, and wherein thereference images are coregistered at least in part, to form a compositehaving a coordinate system, by estimating one or more underlyingtransformation parameters to map the moving image onto the fixed imagecoordinate system.
 9. The method of claim 1, wherein the referenceimages are coregistered, at least in part, using a similarity transformthat uses one or more transformation parameters selected from a groupconsisting of: translation, rotation and scaling.
 10. The method ofclaim 1, wherein the reference images are coregistered by estimating oneor more underlying transformation parameters to map a moving image ontoa fixed image coordinate system, where;${\underset{\theta}{argmin}{F\left( {{I_{F}\left( {x_{F},y_{F}} \right)},{I_{M}\left( {T\left( {x_{M},{y_{M};\theta}} \right)} \right)}} \right)}},$where F is the cost function, and T represents a spatial transformationwith parameters θ.
 11. A system for removing inherent autofluorescenceassociated with a biological material, comprising: a) a processingdevice adapted to analyze a reference image of a biological materialthat exhibits the presence of one or more reference dyes, having a highsignal to autofluorescence ratio; b) one or more filters correspondingto the reference dye and one or more filters corresponding to one ormore information dyes; c) a digital imaging device adapted to acquire afirst set of one or more images of the biological material, inconjunction with the filters corresponding to the reference dye and theone or more information dyes; and further adapted to acquire a secondset of one or more images of the biological material, wherein thebiological material further exhibits the presence of one or more of theinformation dyes, in conjunction with the filters corresponding to thereference dye and the one or more information dyes; wherein the secondset of images comprises a separate image of the biological material witheach of the filters corresponding to the reference dye and the one ormore information dyes; and wherein the processing device is furtheradapted to coregister the reference images; and then remove the inherentautofluorescence, from the images exhibiting the presence of one or moreof the information dyes.
 12. The system of claim 11, wherein at leastone of the reference dyes corresponds to the UV spectrum.
 13. The systemof claim 12, wherein at least one of the dyes that corresponds to the UVspectrum is DAPI.
 14. The system of claim 11, wherein at least one ofthe reference dyes corresponds to the IR spectrum.
 15. The system ofclaim 11, wherein the digital imaging device is adapted to acquire areference image of the biological material using a reference filtercorresponding to a cyan fluorescent protein; and wherein the processoris adapted to coregister the reference images by estimating one or moreunderlying transformation parameters to map the image coordinate system.16. The system of claim 11, wherein the reference images arecoregistered using an intensity-based or a featured based parameters.17. The system of claim 11, wherein the reference images arecoregistered, at least in part, using a similarity transform thatincorporates translation, rotation, and scaling.
 18. The method of claim11 wherein the reference images are coregistered by estimating one ormore underlying transformation parameters to map a moving image onto afixed image coordinate system, where;${\underset{\theta}{argmin}{F\left( {{I_{F}\left( {x_{F},y_{F}} \right)},{I_{M}\left( {T\left( {x_{M},{y_{M};\theta}} \right)} \right)}} \right)}},$where F is the cost function, and T represents a spatial transformationwith parameters θ.
 19. A system for removing inherent autofluorescenceassociated with a biological material, comprising: a) a processingdevice adapted to analyze a reference image of a biological materialtaken using a reference filter corresponding to a cyan fluorescentprotein, having a high signal to autofluorescence ratio; b) one or morefilters corresponding to one or more information dyes; c) a digitalimaging device adapted to acquire a first set of one or more images ofthe biological material, in conjunction with the reference filter andthe filters corresponding to the one or more information dyes; andfurther adapted to acquire a second set of one or more images of thebiological material, wherein the biological material further exhibitsthe presence of one or more of the information dyes, in conjunction withthe reference filters and the filters corresponding to the one or moreinformation dyes; wherein the second set of images comprises a separateimage of the biological material with the reference filter and with eachof the filters corresponding to the one or more information dyes; andwherein the processing device is further adapted to coregister thereference images; and then remove the inherent autofluorescence, fromthe images exhibiting the presence of one or more of the informationdyes.
 20. The system of claim 19, wherein the reference images arecoregistered using an intensity-based or a featured based parameters.21. The system of claim 19, wherein the reference images arecoregistered, at least in part, using a similarity transform thatincorporates translation, rotation, and scaling.
 22. The method of claim19 wherein the reference images are coregistered by estimating one ormore underlying transformation parameters to map a moving image onto afixed image coordinate system, where;${\underset{\theta}{argmin}{F\left( {{I_{F}\left( {x_{F},y_{F}} \right)},{I_{M}\left( {T\left( {x_{M},{y_{M};\theta}} \right)} \right)}} \right)}},$where F is the cost function, and T represents a spatial transformationwith parameters θ.